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(57) Abstract 

A virtual reality medical siinulatton system (1) provides a highly realistic simulation of a variety of medical procedures, particularly 
catheter based procedures such as balloon angioplasty. A catheter interface device (3) tracks a catheter wire (7) in translation and rotation 
and outputs a signal to a computer program mat displays movement of a virtual catheter within a virtual arterial tree including an occluded 
region. The catheter tip is navigated through the arterial tree and into me occluded region, whereupon the ccroputer program generates a 
tactac feedback force mat is output to the cameter mterface device (3) to simulate increased resistance. Once the catheter (7) is properly 
positioned within the occlusion, a virtual balloon catheter can be inflated to remove the occlusion. A variety of special effect enhancements 
areincluded in the simulation, including 3-D imaging, simulated radiopaque dye brfusion, bleeding and atrial pulsation in synchronism with 
an on screen EKG display. 
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COMPUTER BASED MEDICAL PROCEDURE SIMULATION SYSTEM 



HAfTKfiROUND 

The present invention relates to computer-based simulation of medical procedures, 
e.g., surgical procedures. A particular application of die present invention concerns a 
system for providing a realistic "virtual reality" simulation of a cardiac cathete ri zati on 
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Continuing public debate 



pin-pointed 



(CME) and graduate medical education (GME) 



expenses to address in die health reform effort More than $6 billion is spent on CME and 
GME annually. Much of this cost is attributed to expensive training, resources and 
material. The use of virtual reality (VR) in medical education and training can reduce the 
expenses associated with changes in medical technology, as well as provide 

physician training. 

VR technology has been used in a wide variety of fields, including 
architecture, chemistry and biomedicine. The application of VR technology to 



g 



and training, however, is just beginning to be exploited. VR technologies are 1 
more applicable in medicine, in part because of the high costs of traditional training 
resources (lab animals, physican's time, etc.). 

The value of comp u t e r-assisted instruction has long been reco gn ized. During die 
past drcadf , numerous medical centers around the country have designed and used software 
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computers. 



ructional programs also have many 
advantages over tra ditional lecture-oriented learning, e.g., they are interactive and allow 
for self-paced instruction. Many of the programs also incorporate simulations which 
require die user to make patient management decisions. If incorrect decisions are made, 
text describing why die choice was incorrect is provided. 

recently, medical education programs have been created using personal 
graphics for the instruction of anatomy. This type of program provides i ncr ease d 
al value because it gives students die opportunity to quickly access anatomic 
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spending 
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dissecting cadavers (and being exposed to toxic che m i c al preservatives), these programs 
allow students to work in the Kbiary - or at home - and spend as much time as needed to 
learn the material. The programs also provide textbook references so students can 
immediately read about the clinically relevant aspects* 

Although these computer-assisted programs offer many advantages in medical 
education, there are some inherent limitations and drawbacks. For example, these 
programs are confined to two dimensional (2-D) imaging. This limits the scope of 
depicting anatomical orientation, particularly with respect to showing the relation of 
superficial and deeper structures. These programs are similarly limited by lacking the 
ability to rotate in space so that anterior and posterior relationships can be seen. 
Additionally, while highly valuable for anatomic instruction, these programs do not allow 
for practice of actual medical procedures under realistic conditions. 

A system capable of providing a realistic real-time simulation of surgical procedures 
would help reduce die operative risk associated with new technologies in medicine, allowing 
better transference of skills from training to the operating room. A tremendous demand 
exists for enhancing the way that physicians learn new invasive procedures. Appropriate 
education in new medical and surgical procedures is often outpaced by the desire of 
physicians to incorporate a procedure into their practice. 

As one example, until a few years ago, the treatment for coronary artery disease was 
open heart surgery, an expensive and very difficult surgical procedure. As an alternative 
to open heart surgery, percutaneous transluminal coronary angioplasty (FTCA) is now 
generally considered safe and effective to relieve the same conditions that heart surgery did 
just a few years ago. With the recent availability of this new procedure, the training of 
physicians presents a considerable problem. Currently, physician experience depends cm 
the availability of a weekend course and practicing the procedure in an animal lab. 
Unfortunately, two days rarely provides enough practice time. Additionally, animal 
anatomy is quite different than human anatomy. A realistic virtual reality simulation could 
offer the physician the ability to practice hundreds of procedures in a realistic environment, 
and with realistic anatomy, prior to patient contact 

Computerized surgical simulations have the potential for improving surgical 
morbidity and mortality. Studies have shown that for a wide range of diagnostic and 
therapeutic procedures, doctors doing their first few to several dozen cases are much more 
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likdy to make a greater number of errors. This phenomenon has been referred to as the 
"learning curve." Adequate proctoring of learners by experienced surgeons is cumbersome, 
as there are few surgeons experienced enough to proctor their colleagues. It is difficult for 
physicians, particularly those in rural areas, to travel to larger medical centers for training. 
The requirement also places a burden on experts who, already busy with their clinical 
responsibilities, could become overwhelmed with proctoring requests. 

Another potential advantage of VR surgical training is in the standardization of 
surgical training. Such standardization would help insure, e.g., that all board-certified 
physicians are top-notch. Providing training with computer programming would allow 
major medical centers to directly interact with students and physicians in a rural setting. 
By so doing, the credentialing and granting of privileges for traditional procedures, as well 
as newer procedures like laparoscopic surgery, could be objectively provided based on 
standardized skills assessment, regardless of where a surgeon is located. The challenge has 
been to create a surgical simulation that is sufficiently realistic that the motor skills and 
anatomic recognition acquired in die simulation are readily transferred to performance of 
the actual procedure. 

In addition to realistically simulating the involved anatomy, i.e., body parts, a 
realistic computer-based surgical simulator must also provide a realistic surgical 
environment. A key in this regard is to provide the involved proxy medical instruments) 
with a realistic look and feel. This can be accomplished through die use of computer 
peripheral devices that mimic the actual instruments) used by a surgeon to perform an 
actual procedure. In this respect, the closer the proxy surgical instrument is to the actual 
surgical instrument, the more realistic the surgical simulation. 

One computer interface device that has been used in surgical simulations is the 
Immersion PROBE produced by Immersion Corporation of Palo Alto, California. The 
Immersion PROBE comprises a pen-like stylus supported on a light-weight mechanical 
linkage. The stylus is held between die fingers like a medical instrument and is wielded 
dexterously. The Immersion PROBE reports the position and orientation of the stylus to 
a host computer via a standard serial port interface. The Immersion PROBE linkages 
provide six degrees of freedom. Sensors provided at the linkage joints convey spatial 
coordinates (X,Y,Z) and orientation (roll, pitch, yaw) of die stylus to die host computer. 

While die Immersion PROBE is a useful general purpose 3-D c ompu t e r / user 
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interface tool, it does not closely resemble in fed or appearance an actual surgical or 
medical instrument- For example, it's pen-like stylus and six degrees of freedom bear little 
resemblance to the type of elongated flexible member used in a catheterization procedure. 
An additional shortcoming of die Immersion PROBE is that it has no provision for 
providing tactile feedback to the user. Tactile feedback is very important to create a 
realistic feel. 

Hon U.S. Patent No. 4,907,973 discloses a computer-based surgical simulation 
system including a peripheral device designed to simulate a surgical instrument, as well as 
the involved anatomy, i.e., body parts. In particular, the Hon patent discloses a system for 
computer simulation of a balloon-angioplasty operation. Therein, a catheter-like device is 
negotiated through a mock arterial path of a physical internal body part modeling device. 
Sensors on the catheter track the progress of the catheter through the mock arterial path. 
In one embodiment, a vessel constricting simulator within the mock arterial path provides 
a fixed resistance to die progress of the catheter. 

While the Hon system has the potential for providing a more realistic fed of a 
catheter device as compared with the Immersion PROBE, it lacks flexibility for easily 
altering the involved anatomy and procedures. The simulation shown on the computer 
monitor appears to correspond to the structure of the physical model. Providing different 
anatomies, patients conditions and procedures would require additional physical models for 

Another shortcoming of the Hon system is its lack of "special effects" for 
r ealisticall y simulating changes in the virtual environment responsive to operator 
manipulations performance- For example, in the catheter insertion simulation of Hon, 
no provision is made for (1) a realistic simulation of bleeding, e.g., in die event an arterial 
wall is punctured, (2) radio-opaque dye infusion to render the arteries visible under a 
fluoroscope, or (3) catheter balloon inflation and consequent removal of an inclusion. 
Moreover, the Hon system is over-simplified and less than realistic, since no provision is 
made for tracking and displaying catheter rotation in the virtual environment. Importantly, 
in an actual catheterization p roce dur e, rotation of the catheter wire is used, e.g., to direct 
the catheter in the proper direction. 
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SUMMARY OF 




In view of the foregoing, a principal object of the present invention is to provide for 
tiie performance of a simulated medical procedure under highly realistic conditions. 

It is a further object of die invention to provide a high degree of flexibility, thereby 



including actual patient conditions. 

Yet another object of the present invention is to combine with a realistic simulation 
experience, tutorial aspects such as expert assistance and evaluation. 

These and other objects are achieved by a computer based medical procedure 
simulation system in accordance with the present invention. In one such system, first 
memory means are provided for storing image data representative of a virtual internal body 
environment Second memory means are provided for storing image data representing a 
virtual medical instrument extending within the internal body environment Display means 
are provided for displaying images corresponding to the image data stored in die first and 
second memory means. A first computer peripheral device trades die movement of a 
physical member representing die virtual medical instrument, and outputs a signal based 
thereon. First input means are provided for receiving the signal from die peripheral device. 
First c alculating means are provided for calculating position data represe nt ing die position 
of the virtual medical instrument within die virtual internal body environment, based on die 
signal from die peripheral device. First updating means are provided for updating the 
image data representing die virtual medical instrument extending within the virtual internal 
body environment, based on said calculated position data. 

An exemplary embodiment of the present invention is a cardiac catheterization 
simulator used to train interventional cardiologists to perform specific catheterization 
procedures such as angioplasty and stent placement. The cardiologist works with die 
system in the same way that he/she would perform an actual catheterization, Le., by 
manipulating a proxy catheter . The location and movement of the catheter is monitored by 
watching a simulated fluoroscop i c view, and die cardiologist can fed die interaction 
between die catheter and die artery by the force required to move the catheter. 

These and other objects, features and advantages of the present invention will be 
readily appar e nt and fully understood from the following detailed description of prefe r r ed 
embodiments, taken in connection with the appended drawings. 



allowing perform ance of a variety of medical procedures under dl 
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BRIEF DESCRI PTION OF THE DRAWINGS 

Figure 1 is a schematic view of an overall medical procedure simulation system in 
accordance with the present invention. 

Figure 2 is a schematic view illustrating the operating principles of a catheter 
interface device used in the system shown in Fig. 1. 

Figures 3-17 are logic flow diagrams for a program used in die simulation system 
of Fig. 1. 

Figures 18-29 are representative screen displays produced in die simulation system 
of the present invention. 

Figure 30 is a front-left perspective view of the catheter interface device used in die 
simulation system shown in Fig. 1. 

Figure 31 is a rear-right perspective view of the catheter interface device shown in 
Fig. 30, with die cover removed. 

Figure 32 is a front devational view of die internal componentry of die catheter 
interface device, with the electrical connections omitted for visual clarity. 

Figure 33 is a rear devational view of the internal componentry of the catheter 
interface device, with the electrical connections omitted for visual clarity. 

Figure 34 is a right-side devational view of the internal componen t ry of the catheter 
interface device, with die electrical connections omitted for visual clarity. 

Figure 35 is a left-side devational view of die internal componentry of the catheter 
interface device, with die electrical connections omitted for visual clarity. 

Figure 36 is a top-plan view of the internal componentry of the catheter interface 

device. 

Figure 37 is a cross-sectional view taken along a line 37-37 in Fig. 36. 
EEIAILEB DESCRIPTION OF the preferred embodiments 

An overall simulation system 1 in accordance with the presen t invention is illustrated 
in Fig. 1. The system comprises two primary parts: a computer/human interface device 
3 and a computer 5 storing a software program which works in conjunction with the device. 
In die exemplary embodiment described in detail herein, the system simulates a cardiac 
catheterization procedure, and interface device 3 is a catheter interface device. Catheter 
interface device 3 tracks the transbtioual and rotational movement of a proxy catheter 7 
provided in the form of a wire or like elongated flexible object. Catheter interface device 
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3 also provides selective tactile feedback resistance to the motion of the proxy catheter. 
The translation and rotation of the wire are converted into digital signals which are sent 
over line 9 to the simulation program. The virtual reality simulation is presented on a high 
resolution computer monitor 10. The feedback resistance is controlled by digital signals 
11 sent by the program back to the catheter interface device over line 11. 

In addition to die catheter interface device, the system comprises a switching 
me chanism , e.g., an ON-OFF foot switch 13. When activated in one mode, the switch 
triggers the software to simulate a release of radio-opaque contrast material. This allows 
for visualization of die coronary arteries, including die area in die artery that is obstructed, 
as in an actual procedure. In a second mode encountered when the virtual cat h eter enters 
an occluded portion of the virtual artery, die switch serves to simulate a syringe operated 
to selectively inflate and deflate a virtual balloon catheter. Randomly generated or user 
selected patient determinative data will determine how many successive inflations and 
deflations of the virtual balloon catheter are required to successfully remove the occlusion. 
In place of a simple footswitch providing a dual function, separate devices (e.g., proxy 
syringes) may be employed to actuate dye infusion and balloon inflation. Other devices that 
may be incorporated into die various types of catheterization procedures may be employed 
as well. 

For enhanced effect, die screen simulations can be presented in 3-D. This is 
accomplished through the use of a slereographic system, e.g., CrystalEyes available from 
Stoeo Graphics, Inc. of San Rafael, California. This product uses eyeglasses 15 having 
active liquid crystal shutters which open and close sixty times per second. The shutters are 
synchronized to monitor 10 via an infrared emitter 17 mounted on the front of the monitor, 
and a receiver 19 mounted on the glasses. Although not used in an actual angioplasty 
procedure, 3-D imagery provides a more sophisticated examination of the cardiac anatomy 
for these types of procedures. A surgeon's ability to see organs and tissues in 3-D, in 
laparoscopic p roced ur e s , and similar angioplasty procedures, will give physicians an 
enhanced ability to view and manipulate those organs and tissues. 

Digital, compact-disc quality multi-media audio can be used to provide an 
instructional narration in the inventive system, as well as to provide soothing background 
music as is often provided during actual surgery. For this purpose, the illustrated system 
includes a CD-ROM drive 2 1 , an audio mixer 23 (for mixing background music from the 
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CD-ROM drive and a narrative "voice-over" stored on the computer harddrive), and a pair 
of amplified speakers 25* 

The operating principles of the catheter interface device tracking mechanism are 
schematically illustrated in Fig. 2. To obtain translation tracking, a friction wheel 27 
presses against the catheter wire 7. This causes the wheel to rotate when the wire is pushed 
or pulled. 

An optical encoder 29 is attached to the friction wheel for measuring the rotation. 
Rotation measurement is obtained by passing the catheter wire through a semi-tight 
grommet 31. When die wire is rotated, a wheel 33 attached to grommet 31 rotates. An 
optical sensor 35 mounted adjacent to die wheel measures the rotation of wheel 33 and 
outputs a corresponding signal. At the same time, grommet 31 allows the catheter wire to 
slide translationally therethrough. 

A feedback resistance force is achieved by having wire 7 slide through a wire 
pressing mechanism 37 actuated by a solenoid 39. When the solenoid is activated, it pushes 
a plunger 41 against one side of a sleeve 43, pressing die catheter wire 7 and thereby 
making it more difficult to move die same. 

An exemplary catheter based simulation is the simulation of PTCA. As mentioned 
above, PTCA is a technique for relieving partial blockage of arteries in die heart In the 
actual procedure, a wire-like guiding catheter is threaded through an artery in the leg, up 
to the blocked area in the heart The surgeon can see where the wire is going by injecting 
radio-opaque contrast material into the artery, thereby rendering the artery visible on a 
fl uo ro sc op i c (X-ray) display. The physician directs the catheter' s placement by twisting and 
pushing as it is guided through the artery. Once the end of die catheter is in die proper 
position, die surgeon can thread a balloon catheter onto die guiding catheter. Once die 
balloon catheter is properly placed, the surgeon can inflate the balloon to open die occluded 
portion of the artery so that regular blood flow is restored. More recently, a self-guiding 
balloon catheter have been used to perform the procedure, thus eliminating the preliminary 
step of placing a separate guiding c atheter . 

The exemplary simulation system shown and described in detail herein proceeds very 
much like an actual PTCA procedure, particularly (me using a self-guiding balloon 
catheter. Additionally, a number of simulation enhancements are provided, as described 
in detail herein below. It will be appreciated that the principles of the invention could be 
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applied in order to create a double 



insertion simulating more closely the more 
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balloon 

The software monitors the translation and rotation of cat he ter wire, as the 
cardiologist manipulates it Using this information, the program tracks the progress of the 
catheter through a wHflterf arterial tree forming part of a 3-dimensional internal body 
environment The simulation program uses geometry data which can be created for each 
specific procedure (and virtual patient). The location and orientation of catheter 7 is 
convert e d into an image by the program, which image is displayed on computer monitor 
10. The image has the same fluoroscopic appearance as a cardiologist sees during an actual 
procedure (but wife the optional enh 

displays of Figs. 22-26). In addition, other views which are not possible during a 

e provided by die simulation program to enhance the instructional value of 
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the simulation (see Figs. 27-29). Depending on the position and orientation of die virtual 
catheter within the virtual arterial tree, the simulation program may calculate and output 
to catheter interface device 3 a resistance fi^ signal so that the cardiologist c^ 
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he presentation of whic 
, or selected by the user. 

system can si nH ' Httc p rocedural emu 
wire is moved too abruptly, this triggers the 
[, and associated bleeding (see Fig. 25). If die 



also 



virtual balloon catheter is inflated to an excessive amount, a realistic simulation of ball 

rupture occurs. 

As seen in Figs. 21-29, a simulated electrocardiogram 

be displayed on die monitor 10. Provision is made for 
"abnormal" to "normal" during the course of die si mul a t i o n. Provision can also be made 
for the random occurrence of abnormalities such as atrial fibrillation. Addition al realism 
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is provided by simulating pulsation of the arterial tree (and the catheter therein) in 
synchronism with the EKG. 

Additional abilities that can be provided by die simulation system include, among 
others, interactive control of the point of view, recording of the procedure, and monitoring 
by an expert assistant Changing the point of view is important. For example, viewing the 
arterial tree from different perspectives allows the cardiologist or student to better learn the 
^dimension* 1 iriatwmchipt nf fhg arterial branches. This is difficult to master in practice 
since only a 2-dimensional projection of the 3-dimensional geometry is available in die 
fluoroscopic view used during an actual angioplasty procedure. Hie progress of the 
cardiologist can be recorded and played back for review and evaluation. A virtual "expert 
assistant* can be included to monitor the actions of the cardiologist during die simulation 
and can provide information such as warnings and explanations. 

Many different catheter-based procedures can be simulated with die inventive 
system. The specific procedure is created by the data used in die interaction aspects of the 
simulation program. Thus, the same catheter interface device can be used with many 
different p roced ures whDe die simul a tio n program is c us t omi zed for each procedure. This 
enables the catheterization simulation system to be used to create training environments for 
many di ffer e n t clinical procedures and patient conditions. The patient conditions can be 
simulated or actual With respect to actual conditions, those conditions could be taken from 
representative past patients for use in general training, or could be taken from a current 
patient for use in a pro-operative surgical rehearsal. 

The logic flow of die exemplary FTCA simulation program (and representative 
screen displays thereof) are now described in detail, with reference to Figs. 3-29. 

Figure 3 depicts three primary modules of the main program: error checking 45, 
case presentation 47 and calibration 49. In error checking module 45, features and function 
of die system are diagnosed for presence and functionality. As previously mentioned, a 
stenographic 3-D viewing system can be utilized by die system to enhance die simulation. 
To determine if the stenographic transmitter is present , a call to the device is made at step 
49. If die transmitter is present, the video is adjusted for three dimensional perception at 
step 51. If die transmitter is not present, the stereographic viewing function is disabled at 
step 53 and the program proceeds without die visual enhancement The system may 
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alternatively utilize other types of visual perception enhancement devices, such as 
headmounted displays, and projection devices (including holographic projection devices). 

As previously mentioned, the system utilizes a catheter interface device 3 that 
provides for direct human interaction with the computer simulation. The presence and 
functionality of catheter interface device 7 is determined at step 55. If the device is not 
delected or if it fails a diagnostic routine, the error is reported via the display and the 
simulator program will exit to die host operating system (step 57). 

Given that catheterization interface device 3 passes die previous error checking 
routines, die program next proceeds to the detection of fbotswitch 13 used for dye infusion 
and balloon inflation (step 59). If die system foils to detect the fbotswitch, an error is 
visually reported and die program exits to the host operating system (step 61). 

To synchronize the program polling rate, simulation timing functions, and threshold 
arrays, the program implements a routine to assess die speed of the host computer system 
(step 63). All synchronization variables are output to global data buffers to be utilized by 
time dependent subroutines, array determinant threshold subroutines, and polling count 
determinate subroutines within die simulation program. Next, the program proceeds to a 
case presentation module 47. 

In case presentation module 47 , a patient case and its relative data are retrieved (step 
65) via a random selection subroutine 67, from an cm-line database 69. This stage may be 
replaced or supplemented by a routine allowing die user to select a specific patient case 
from the case files, or to input data from a selected patient case not in the stored files. The 
latter would be particularly useful for conducting a preoperative rehearsal. The case 
specific data which is stored in memory in the preferred embodiment can include EKG data 
(normal and abnormal), occlusion location data and angioplasty success data, i.e., the 
required number of successive balloon inflation cycles required to remove the occlusion. 
(The angioplasty success data is artificial since in an actual procedure the number of balloon 
inflations required to remove an occlusion typically is not known in advance.) The case 
determinant variables are set in step 71. 

Patient information is delivered via a multimedia presentation at step 12. This may 
include a patient history screen as shown in Fig. 18, an arteriogram as shown in Fig. 19, 
and an indication of hemodynamics as seen in Fig. 20. In addition to displaying patient 
information on die screen, corresponding or supplemental information is conveyed by a 
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digitally recorded voice provided through the multi-media audio system. If fbotswitch 13 
is activated at step 75, die presentation will proceed ("fast forward") to die next topic (step 
77). So long as the program polling does not detect an activated fbotswitch, die program 
will proceed in a linear fashion through each topic, until the last topic is readied (step 79). 

Before die simulation actually begins, the catheter interface device is calibrated 
within calibration module 49 (step 81). To calibrate catheter interface device 3, die 
catheter wire 7 is synchronized with the program by the user. The user is in struct ed by the 
program to pull the guide wire out while a status bar on die screen display of monitor 10 
decreases to indicate the approach of the catheter starting point. This calibration is required 
for die program to ascertain the point of origin for screen representation and translation 
tracking. Rotation degree is also set to a starting position at this time. 

Referring now to Kg. 4, upon completion of the calibration routine, die program 
will proceed to a catheter placement simulation module 83. At this stage, die screen 
displays a simulated fluoroscopic (X-ray) view (see Fig. 22), wherein die patient's rib-cage 
84 is clearly seen. The arterial tree remains invisible until a subsequent dye infusion 
routine is executed. Once the dye infusion routine is executed, die arterial tree 86 including 
an occluded region 88, and the virtual catheter 90, become clearly visible (see Figs. 23-29). 
The background image can be created by digitizing an actual flu or oscop i c image and, if 
desired for clarity, by modifying die digitized image data to remove unwanted structures 
and "noise". In the present embodiment, this background remains inanimate throughout die 
simulation (as it generally would during an actual procedure). As also seen in Figs. 21-29, 
the screen presents along its bottom border a moving EKG waveform 92, based on the 
stored patient data ■ 

In catheter placement simulation module 83, a timer subroutine 85 simulates real 
time by counting units that are determined by the polling rate time adjustment step 63. 
Referring to Fig. 5 showing the timer subroutine, die timer unit is defined as an integer 
with a value of I. If die current sum timer value is greater than 1 at step 83a, then the 
value is increased by 1 unit at step 85b. The value is set to 1 upon initiation of the timer, 
at step 85c. 

Referring back to Fig. 4, die program proceeds to an EKG data determination 
subroutine 87. Referring to Fig. 6, if an occlusion (blockage) is indicated in die case 
specific arterial dataset (blockage=l) at step 87a then die program will check to see that 
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the abnormal EKG data is stored in the program memory buffer (step 87b). If it is not yet 
stored, the case determinant data is retrieved at step 87c from the EKG database 87d. The 
EKG data base 87d can provide sinus rhythm patterns as well as atrial fibrillation data, and 
other case determinate cardiac events. The buffer space is given a local definition, Le. 
CASE_ABNORMAL at step 87e, and the EKG data is placed in a common buffer (step 
87f). Alternately if the correct buffer name is found at step 87b, then die program will 
proceed out of the subroutine. 

Conversely, if it is determined that a blockage does not exist at step 87a, then the 
program will check to see that the EKG data is set to CASE_NORMAL at step 87g. If it 
is, die program will proceed out of the subroutine. Otherwise, the CASE_NORMAL data 
is retrieved at step 87h from die EKG database 87d. The buffer name is set to 

at step 



CASE_NORMAL at step 87i, and die data in the EKG data buffer is overwril 
87f. The program provides the onscreen EKG simulation based on this data in die EKG 
Buffer, which is derived from die stored case data. 

Referring momentarily back to Fig. 4, die onscreen EKG display is produced and 
updated within subroutine 89. Referring to Fig. 7, the update EKG subroutine will check 
to see that die EKG data is translated into a usable array at step 89a. If an array is not 
detected, die EKG data is retrieved at step 89b from die EKG data buffer 89c and sorted 
i array. The array is processed at step 89d by translating the value from the EKG 
array 89c, to a floating-point number between 0 and 1. A value equal to 75% to 
80% of die highest value in the translated array is set as an audible beep threshold value 











7: 
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at step 89e. The adjusted array is next defined as the spline deformation array at step 89f . 
If it is determined at step 89a that the spline deformation array has already been set, die 
subroutine will bypass die array definition process. 

Parf> time die program polls update EKG s creen subroutine 89, die spline 
array is advanced one place holder (step 89g). A range from die array is then 
rm the EKG screen spline at step 89h. The real patient data may be plotted and 
mi decay can be simulated. If it is determined at step 89i that the apex of the 



defi 



spline is at or above the defined beep thr 
the sound system 89), at step 89k. 

Tte simulation can include, as a simulated oompli 



value, thai an audio beep is generated by 



randomly generated periods 
in the screen display of Kg. 
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21, wherein the surgeon is instructed to stand-by until normal sinus rhythm resumes. At 
this point in an actual procedure, a drug would be administered to the patient Such a step 
could be included in the present simulation, e.g., using footswitch 13 as a trigger 
mechanism. Alternatively, the simulation can be simplified by simply notifying the surgeon 
that the drug has been administered. Another feature that can be provided is a graphic time 
dock 94 (as shown in Fig. 21) notifying die surgeon of the time remaining before the drug 
takes effect (and sinus rhythm is resumed). 

Referring again to the main program of Fig. 4, at subroutine 91 an arterial pulse is 
realistically simulated by synchronizing arterial and catheter movement with the EKG pulse 
data. Referring to Fig. 8, subroutine 91 will retrieve a spline deformation array at step 
91a, from a spline deformation buffer 91b. The spline deformation array is the data used 
to represent the virtual arterial tree that will appear upon dye infusion. The configuration 
of the arterial tree can be adapted from commercially available 3-D graphics packages (such 
as from ViewPoint Data Labs of Qrem, Utah) or can be independently generated based 
upon physical models or actual anatomies. Offset data points are extrapolated from the 
array data at step 91c. The geometry is offset (redrawn) and die screen is updated at step 
91d. The program then proceeds to step 93 of the main program (Fig. 4), where the status 
of footswitch 13 is checked. At this stage, footswitch 13 is used to initiate the infusion of 
contrasting dye (subroutine 95). The default position of the switch is OFF and the program 
will simulate rapid dissipation of the dye into the bloodstream (as would occur in an actual 
procedure) when the footswitch is released. At step 93, a check is made to see whether the 
footswitch has been depressed. If not, dye infusion subroutine 95 is by-passed. 

When the footswitch is depressed, die program proceeds to the dye infusion 
subroutine 95. Referring to Fig. 9, dye infusion is activated at step 95a, whereupon a 
ramp constrained, random flow of values is streamed into an infusion value definition (see 
95b) used to determine the shade or concentration of the dye. By altering die value of die 
dye concentration at various points, die shade variations simulate die mixing of blood and 
dye. By ramping the value stream, simulated dye that is infused behind dye infused earlier 
will appear less diffused, thereby looking to be of higher concentration. By randomizing 
die shade determinative value stream as it is ramped upwardly, it is possible to create a 
strikingly realistic simulation of one fluid On this case dye) differing into another fluid On 
this case blood). The constraints on the ramped random value stream 95b can be adjusted 
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to provide the appearance of different fluid viscosities and rates of diffusion. Alternatively, 
a particle system or procedural material map may be used. 

The path of the dye is mapped (at step 95c) in accordance with fee artery data set 
95d. The flow of fee dye is mapped progressively from the top of the artery down through 
fee branches. When the footswitch is released, the mapped contrasting dye is reverse 
mapped to simulate diffusion in a downward direction, as if blood were flowing from above 

and "washing" fee dye from fee site. 

If it has been determined at step 95e feat a puncture in fee arterial wall has been 
mapped cm fee arterial geometry, and footswitch 13 is in the ON position, fee puncture 
origin 95f is mapped at step 95g, to simulate bleeding from fee puncture into fee 
surrounding body cavity. Such bleeding is shown at 96 in the screen display of Fig. 25. 
If no origin is detected, fee program will proceed. As will be described, a puncture is 
caused to be mapped onto fee geometry when fee user moves the catheter wire 7 of fee 
catheter interface device 3 too abruptly. 

Hie next step in fee main program is a catheter movement sampling subroutine 97. 
Cafeeter interface device 7 includes an electronic interface having a software driver for 
interpreting fee signals produced by the catheter wire tracking sensors. As shown in Fig. 
10, fee data feat is provided by fee driver is sampled at step 97a, at a rate feat has been 
determined by the program at step 63. Longitudinal translation 97b and rotation 97c of fee 
wire through fee catheter interface device are processed to determine fee wire location 
relative to the calibration point position established in step 81. The Z~axis (corresponding 
to fee cafeeter insertion depth) and the rotation data are placed in an array buffer in step 
97d. 

The program then proceeds to subroutine 99 for determining fee location and 
orientation of fee virtual cafeeter. As shown in Fig. 11, this is computed at step 99a from 
fee Z-axis and rotation value array stored in array buffer 97d , a geometry offset value 9% , 
and the constraints provided by fee artery data set 99b. By taking account of a geometry 
offset value (which is a product of fee arterial geometry data and fee EKG data), a 
simulated flutter or vibration of fee cafeeter in synchronism with fee arterial pulse is 
achieved. Tbt cafeeter location and orientation data are placed in a data buffer at step 99d 
for use in a subsequent cafeeter update subroutine. 
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With the arterial tree illuminated by simulated fluoroscopic dye, the catheter is 
ed through the arteries toward the occlusion region. Figs. 23-26 show the 



can be seen in these screen displays, virtual catheter 90 has an angled or bent end portion 
98. This allows for steering the catheter down different pathways, e.g., brandies in the 
virtual arterial tree, based upon die rotational orientation of die ca th eter wire. 

To determine whether the catheter has entered occluded region 88, within subroutine 
101 it is determined if the spline origin (representing die catheter end) is within the 
blockage range stored as part of die case specific data. Referring to Fig. 12, the blockage 
range data 101a, and catheter location definition 101b are compared at step 101c. A local 
value of 1 is output at step lOld if the comparison determines that die catheter location is 
within the blockage range. Conversely, if the comparison in d ic ates that the catheter is not 
within die occlusion region, a value of 0 is passed at step lOle. 

If subroutine 101 returns a value of 1, then the program will output a signal to the 
catheter interface device to impart force feedback (a.k.a. haptic feedback, resistive 
feedback) to die cpf***-*- wire (at step 103). The force feedback signal can be provided at 
a single preset level, or could be provided in gradations dependent on the location of die 
catheter and die cross-sectional dimensions of the occluded region. Then, the program 
proceeds to a balloon inflation simulation module 105. If subroutine 101 returns a value 
of 0, die program will proceed to subroutine 107 for checking whether die catheter has 
punctured the arterial wall. 

Puncture simulation in die illustrated program is based cm a de t ected velocity of 
catheter wire 7 within the catheter interface device 3. This provides a realistic simulation 
in that during an actual procedure an abrupt movement of the catheter can often result in 
an arterial puncture. likewise, in the simulation, if the catheter wire is moved abruptly by 
die user, die velocity of die catheter exceeds a preset level and thereby triggers the 

simulation of a puncture. 

Referring to Fig. 13, in puncture detecting subroutine 107, die program will first 
check at step 107a for the presence of a previously defined puncture origin 107b. If a 
puncture origin 107b does exist, then the program checks at step 107c whether an audio 
response (e.g., verbal announcement of die error) through die mu l t i m edia sound system 
107d is completed. During die audio response, the program continues to loop, resulting 



progr 
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in the simulation of bleeding from the puncture origin (see 96 in Fig. 25), due to execution 
of dye infusion subroutine 95. At this time, a flashing screen message may also appear to 
alert the user of the procedural mistake. Following completion of the audio response, the 
simulation proceeds to termination step 107e, whereupon the screen display can be frozen 
to preserve the unsuccessful result for evaluation. 

If a puncture origin is not detected at step 107a, the presence of a previously logged 
velocity variable is determined at step 107f. If none is detected, the current time 107g is 
placed in a local variable definition (bufer A) at step 107h and the program proceeds. If 
a previously recorded value is detected in step 1 07f , then that variable is moved to a second 
buffer (buffer B), in step 107i. The current time 107g is placed in the primary buffer (A) 
at step 107j. The time between events is then calculated at step 107k. If the calculated 
elapsed time is determined to be below a preset value at step 107e, this indicates a velocity 
in excess of the velocity constraint Under this condition, the puncture origin 107b is set 
to the current catheter location (step 107m). The program will then activate the audio 
response to the puncture event (step 107n), and program control returns to the main 
program. If the time between events calculated in step 107k is found Cm step 107e) to be 
within the defined constraints, the program will proceeds to the next stage without a 
puncture origin having been defined. 

An alternative method of detecting a puncture would be to utilize a collision 
detecting component, whereby a puncture would be simulated upon an impact of the virtual 
catheter tip a gainst a wall of the virtual artery. 

In the next main program step (subroutine 1 1 1), the catheter position and orientation 
are updated. The catheter is represented by a spline generated from data points. Referring 
to Fig. IS, to determine the length of the spline, die starting point (i.e. catheter origin 
111a), and the current location of die catheter 111b, are retrieved and processed in step 
111c. In step Hid, the spline length definition computed in step 111c is combined with 
geometry offset data Hie. The curvature of die spline (Le., spline deformation) is derived 
from the geometry ofifcet data 1 1 Id , which is a product of the artery geometry and the EKG 
pulse information. Ifa puncture origin is present (11 If), the path of the spline will not be 
constrained by die geometry map at that point, thus allowing the catheter to pass through 
the puncture, eg., as seen at 96 in the screen display of Fig. 25. 
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The rotation of the catheter wire is reflected by the position of the bent or angled 
wire end portion of the virtual catheter. The catheter is pointed or navigated on the path 
chosen by the user by rotating the wire (and bent end portion thereof). If it is determined 
in step lllg that the end of the catheter is located at a branch in the arterial data set lllh 
(as seen at 100 in the screen display of Fig. 24), the program will determine the direction 
of travel or trajectory at step llli based upon retrieved rotation data lllj. Finally, the 
spline is deformed and redrawn in step 111k, and program control proceeds. 

Referring back to Fig. 4, when the catheter is determined to be within the occlusion 
range at step 101, and after force-feedback is activated at step 103, the program proceeds 
to balloon inflation module 105. Therein, the program will first magnify the occlusion 
region and change the viewing p erspec ti ve (step 113), as seen in die screen displays of 
Figs. 27-29, in order to enhance visualization of the arterial structure and the effect of the 
procedure thereon. Contrasting dye is automatically activated, the visual components are 
adjusted to die new per sp e cti ve, and the polling cycle is redirected to loop within the 
balloon inflation module. The required program components are replicated within this 
stage, i.e., catheter update 111, timer 85, EKG data 87, arterial pulse update 91, and 
footswitch status check 93. Within die balloon inflation module, footswitch 13 bec om es 
operative to activate balloon inflation rather than dye infusion. If step 93 determines that 
the footswitch is activated, the program proceeds to a balloon inflation subroutine 115. 

Referring to Fig. 14 showing balloon inflation subroutine 115, if the footswitch is 
depressed so that a value of 1 is d e tected at step 115a, then the program proceeds to step 
115b wherein it is determined whether an inflation status buffer 115c has been previously 
established. If the buffer has been established, die value within die buffer is increased by 
one unit, at step 115d. The value from die inflation status buffer is then used to deform 
(inflate) die balloon tip 102 of the virtual catheter (step 115e) and then to correspondingly 
deform the occlusion geometry outwardly (step 115f). These effects are seen clearly in the 
screen display of Fig. 28. 

If it is determined at step 115b that an inflation status buffer has not yet been 
previously established, then one is created at step 115g. The value is set to one unit at step 
115h, and the value from inflation status buffer is then used to deform (inflate) die balloon 
tip of the catheter (step 115e), and then to correspondingly deform the occlusion geometry 
outwardly (step U5f). 
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If the fbotswitch value is not equal to 1 at die time step 115a is periled, then the 
program will determine at step llSiif an inflation status buffer 115j has been previously 
established. If it has not, the program proceeds to loop within the main balloon inflation 
module 105. If it has, the buffer is reduced by 1 unit at step 115k, thus serving to 
decrementally deflate the balloon catheter. If the buffer is reduced to a value of 0 (step 
1151), effectively deflating the balloon (as seen in Fig. 29), the buffer is released from 
memory at step 115m. If the condition exists that the buffer has been reduced by 1 unit 
(step 1 15k) but has not been reduced to a value of 0, then the decremented value from die 
inflation status buffer is then used to deform (in this case deflate) the balloon tip of the 
catheter, and then to correspondingly deform die occlusion geometry, at steps 115e and 

115f . 

On exiting balloon inflation subroutine 115, program control proceeds to a balloon 
rupture subroutine 117. A threshold value for inflation is set upon the case specific arterial 
dataset retrieval. This threshold should be equal to the value of units required to fully 
deform the blocked region of the artery so that it is of healthy dimensions. It is possible 
also to introduce die threshold at a lower value to provide for a risk management 
educational dement to the program. Referring to Fig. 16, if the value retrieved from die 
inflation status buffer 117a is determined to exceed die threshold value at step 117b, then 
die balloon volume is set to 0 units (step 117c), abruptly deflating the balloon (as seen in 
Fig. 29) and thereby simulating balloon rupture. This action can be buffered over time to 
deflate at a visually correct rate. Various events such as fragmentation and vascular 
damage may he added to this scenario to enhance die program's realism. When a rupture 
has occurred, die rupture audio response 117d is activated at step 117e, and the program 
then proceeds to a final summation of die simulation step 117f. 

The desired outcome (objective) of the simulation is clearance of die occlusion and 
restoration of normal sinus rhythm. If balloon rupture is avoided in subroutine 117, 
process control proceeds to subroutine 119 for determining when die occlusion has been 
successfully cleared. Referring to Fig. 17, the program will detect whether the balloon has 
been allowed to inflate sufficiently to cause complete deformation of the occlusion range 
cross-sections to die proper (corrected) diameter. If so, an occlusion value of 0 will be 
detected at step 119a. If the cross-sections have not been fully corrected, then the program 
will continue to loop. 
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In an actual procedure, removal of a blockage typically requires repeated balloon 
inflatio n cycles. Similarly, in the simulation, repeated inflation cycles are required to 
successfully clear die occlusion. The required number of successive inflation cycles is 



deten 



mi 



by a success value definition which is part of the retrieved case specific data. 
For each polling cycle wherein it is determined thai the blockage has been fully deformed 
by the balloon to the corrected cross-section, a count value is incremented at step 1 19c and 



a comparison is made at step 1 19d to see if the success value definition has been readied. 
If so, the program proceeds to a successful concluding summation at step 119e. Hie 
concluding summation may, eg., include an announcement of the successful result, and 
provide a display of the elapsed time of the procedure. If the success value has not been 
readied, the program continues through the polling loop. Importantly, because of the 
inflation time limitation imposed by die balloon rupture subroutine 117, a user cannot 
achieve & successful clearance of the occlusion by simply maintaining the footswitch in the 
depressed (ON) position. Rather, it is necessary for the user to successively inflate die 
balloon and allow it to deflate (via die decrementing function of step 115k) at least several 
times, as in an actual procedure. 

In die preferred embodiment, die software is programmed using die industry 
standard language C++ and OpenGL. The use of these two languages has several 
advantages: 

• ability to port the software to different hardware platforms. OpenGL has 
been licensed by many different workstation manufacturers including 
Windows NT, ATT, DEC, and DOS. OpenGL is an open, documented 
standard for describing 3- dimensional interactions with objects. 

• ability of software to adapt to upcoming hardware innovations. OpenGL 
handles most graphics routines, and its use is scaled to the particular 
hardware that is available on die computer. 

• compatibility with Nintendo/Silicon Graphics, Inc. hardware arrangements, 
as wdl as set-top boxes intended for interactive television. Hie availability 
of inexpensive platforms for VR will open opportunities for use in third 
world countries, rural settings and individual homes and offices. 

In order to ma H?™ real-time frame rates (30 fps), it is important to limit picture 



detail to the essential portions of the screen. The limited resource is die graphics rendering 
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speed (the number of polygons per second), and the software must know where to best 
allocate this resource to determine what features being displayed should have the most 
detail. In the exemplary cardiac catheterization simulation described herein, die real time 
graphics are limited to the arterial tree and die catheter therein; die remaining background 
of the simulated fluoroscopic view is inanimate (as it would be in an actual procedure). 
With this arrangement, real time rates are readily attainable. 

Specific attention will now be given to the preferred system hardware for the above- 
described cardiac catheterization simulation system. There are a number of different 
hardware platforms that can be utilized for VR applications. Current configurations 
include: 

High end systems . VR can be run on systems such as die Silicon Graphics, Inc. 
RealityEngine. These are costly and logistically the most difficult to transport due to their 
large size. The Silicon Graphics Indigo 2 Extreme workstation has the graphics and 
computer power necessary for many VR applications, and is less costly. 

Personal computer systems . Although the current generation of personal computers 
could be used for some limited VR applications, they lack the specialized hardware found 
in graphics workstations which facilitates die production of real-time complex 3-D 
manipulation of objects. Personal computers are optimized for 2-D graphics, and as a 
result, they are ideal for playback of digitized linear sequences which can be obtained from 
3-D graphics systems (VR). Personal computers are also well suited for the display of 2-D 
representations of 3-D phenomena. 

The angioplasty application of die present invention was developed cm a Silicon 
Graphics Indigo 2 Extreme workstation. A complete hardware specification for this 

application is set forth below: 

• Silicon Graphics Indigo 2 Extreme graphic hardware (MIPS R4000/100 
Mhz, 64MB RAM, 1GB hard drive, 19" SGI monitor, CD ROM drive, 
DAT tape drive. 

• stenographic equipment (eg., CrystalEyes eyeglasses and emitter + cable). 

• audio equipment (one pair of stereo power speakers, Mackie 1202 (or 
compatible) mixer or another pair of power speakers. 

• audio CD for providing continuous background music (narrative "voice- 
over" can be recorded onto hard drive). 
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• miscellaneous (two stereo mini-male to RCA male cables, six RCA female 
to 1/4" male phone jade, power strip). 

• catheter interface device (as described in detail hereinbdow). 

• foot pedal switch peripheral, e.g., as is available from Immersion 
Corporation* 

In the preferred embodiment, die catheterization simulation system of the present 
invention employs a virtual catheter interface produced for use in applicants' system by 
Immersion Corporation. This device is now described in detail with references to Figs, 30- 

37. 

Catheter interface device 3 tracks the movement of wire 7 in a 3-D space, with die 
movement of die wire constrained to two degrees of freedom. This provides a realistic 
simulation of an actual catheter insertion procedure, wherein once the catheter is inserted 
into a patient, it is limited to two degrees of freedom. More particularly, within a patient's 
body, the catheter wire is constrained to translation (in-out motion) and rotation. 

The catheter wire may be formed of a variety of known materials, e.g., metal or 
nylon. Generally, die gauge should not be bigger than 050 (0.050* diameter), and the 
insertion length should be at least 14". The catheter wire can be an actual catheter wire, 
such as is commercially available, e.g., from Cordis Corporation of Miami Lakes, Florida. 
An actual catheter should be modified such that any tool at die end (such as any cutting 
edges or balloon) is removed leaving only the handle and the shaft 

Catheter wire 7 can include an end mounted handle or "grip" portion 121. The 
"grip" portion can be any conventional device used to manipulate a catheter. Alternatively, 
die ratf«*t^r shaft could be gripped directly. While the particular embodiment described 
herein comprises a wire, it should be understood that other catheters and catheter-like 
members could be used as well, including particularly elongated flexible tubular catheters. 

An electronic tntr**** 123 serves to electrically couple the force feedback actuator 
and motion sensors contained in catheter interface device 7 to die computer 5. An 
electronic interface that is particularly adapted for the present invention is commercially 
available from Immersion Corporation. This is basically die same electronic interface 
designed for use with Immersion's PROBE. Unlike the PROBE setup, a serial interface 
is configured to be bi-directional, so as to allow for transmission of a tactile feedback signal 
from the computer to the force feedback actuator. The electronic interface has six channels 
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corresponding to the six degrees of freedom of the Immersion PROBE, However, with the 
present invention, the electronic interface requires use of only two channels for the tracking 
of catheter wire 7, since the catbrtfT wire is constrained to two degrees of freedom. A 
third channel is used for transmission of the tactile feedback signal. 

interface includes a software driver. The update speed of the device 
I ?lffTy — this speed cannot be exceeded with a single serial port. If 
iry to achieve a cleaner •feel," die device can be connected with two serial ports 
running simultaneously , whereby a 2X speed improvement is obtained by doubling the data 
transmission capability. Other methods of improving interface speeds between the catheter 
interface device and the computer include standard and high speed ethemet connections, use 
of SCSI or direct memory access parts (DMA), or a variety of more rapid standard 
interfaces. In addition, dedicated hardware to facilitate input/output can be used, such as 
boards for rapid data acquisition. 

The operative components of catheter interface device 3 are mounted on top of the 
housing of electronic interface 123 within a separate metal housing 125 having a removable 
cover 127. A rubber-like tubular member 129 is used to assure free passage of wire 7 
through the housing cover 127. Housing 125 could instead be modeled after die external 
bodypart(s) located within die vicinity of die catheter insertion location , eg., a leg or arm, 
in order to render the simulation components more "transparent" to die user. 

The operative components of catheter interface device 3 comprise a block assembly 
131 forming a through-passage 133 (see Fig. 37) and mounting locations far a first input 
transducer (sensor) 135 that senses the translational movement of wire 7 and produces a 
corresponding electrical signal for input to computer 5 , an output transducer (actuator) 137 
Oat imparts a resistance force to wire 7 responsive to a tactile feedback signal received 
from computer 5, and a second sensor — a rotational transducer — 139 for tracking the 
rotation of wire 7. 

Block assembly 131 comprises a plurality of blocks farmed of aluminum or other 
lightweight material, such as plastic. The blocks may be cast, molded, and/or machined 
as a monoblock members, with the aforementioned sensors and actuator attached thereto 

and/or inc or por a ted therein. 

A variety of transducers, readily available in the commercial market are suitable for 
use in die present invention. For example, die input transducers (sensors) can include 
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encoded wheel transducers. 



potentiometers, optical 
(actuator) could be a stepper motor, servo motor, magnetic particle brake, friction brake, 
pneumatic actuator, etc. Hybrid or bi-directional transducers that pair input and output 
transducers together could also be used in place of separate sensor and actuator devices. 
A simple bi-directional transducer that could be used is a permanent magnet electric 
motor/generator. 



It should also be noted that the present inv 



••Hi mi 



can 



•ion 



absolute and 



relative sensors. An absolute sensor is one in which the angle of the sensor is known in 



absolute terms, such as with an analog j 



Relative sensors only provide 
relative angle information, and thus require an initial calibration step, as is provided in the 
present simulation system. 



In the preferred embodiment, actuator 137 is a solenoid capable of producing a force 
output of at least 5 ounces for imparting tactile feedback in die form of passive resistance 
(frictional drag) on wire 7. As best seen in Figs. 35 and 37, a solenoid plunger 141 has 
attached to its end a lower pressing plate 143. Lower pressing plate 143 engages wire 7 
against an upper pressing plate 145 by applying a force in a direction substantially 
perpendicular to the translation direction of wire 7. This produces a frictional force along 
the translation direction of wire 7. Such solenoids, controllable to a thousand or more 
selectable resistance farce values, are available commercially from, eg., Guardian Electric 
of Woodstock, Illinois. It will be appreciated that other actuator devices may be employed 
in die present invention. For example, the actuator could be a pneumatic or hydraulic 
device which applies a force to the wire. It will be appreciated by those skilled in the art 



that the choice of electromechanical, electromagnetic, pneumatic or hydraulic actuation will 
depend largely on the required response time, and die cost and complexity of the device. 
Electromechanical/electrons transducers are generally preferred as they typically have 
a fast response time, are low cost, and are simpler and smaller than hydraulic and 
pneumatic devices. 



One or both of upper and lower press plates 143, 145 can comprise a resilient 
(brake) pad 147 (shown attached to upper press plate 145), for providing a smooth 
braking action to wire 7. Materials appr o pri ate for the resilient brake pad will be apparent 
to those of ordinary skill in die art 



• 24- 

SUBSTITUTE SHEET (RULE 26) 



WO 96/28800 



PCT/US96/02317 



As best seen in Figs. 36 and 37, the translation sensor incorporates a Motion wheel 
149, which wheel is mounted on a rotatable shaft 151. The friction wheel engages wire 7 
with a normal force such that translation of wire 7 causes rotation of shaft 151. As best 
seen in Fig. 35, shaft 151 is coupled to a disk 153 made of clear plastic material and having 
a number of dark radial bands 155 formed on its circumference. As best sera in Fig. 36, 
a photodetector pair 157a,b including a light source and a detector is positioned on opposite 
sides of disk 153 in alignment with bands 155. As disk 153 rotates about its central axis, 
bands 155 ahernatingly cause light to impinge ami not impinge upon the detector, thereby 
creating an electronic signal which is passed to electronic interface device 123 over a bus 
wire 157. In die illustrated preferred embodiment of catheter interface device 3, 
translational movement of the wire is tracked with a resolution of at least 0.002". One 
suitable type of sensor is an optical encoder wheel type input transducer (Model SI) 
marketed by U.S. Digital of Vancouver, Washington. The same basic type of sensor can 
be used for rotation sensor 139. 

In the preferred embodiment, the rotation sensor 139 includes a disk 159 rotatably 
coupled with wire 7. Disk 159 is preferably made from a clear plastic material and has a 
number of dark radial bands 160 formed on its circumference. A photodetector pair 161 
including a light source and a detector is positioned on opposite sides of disk 159 in 
alignment with bands 160. As disk 159 rotates about its central axis, the bands 
altematingly allow light emanating from die light source to impinge and not impinge upon 
the detector, thus producing an electronic signal which is also passed to electronic interface 
device 123 over bus wire 157. A second photodetector could be provided so as to allow 
determination of the rotation direction, as is well known to those skilled in the art. 

The engagement between wire 7 and die central aperture of disk 159 is preferably 
accomplished by the formation of a friction seal. This can be provided by a resilient (e.g., 
rubber or plastic) grommet 163 mounted within the hub of disk 159. Grommet 163 
receives wire 7 with a send-tight fit causing disk 159 to rotate coaxially with the wire, 
while at die same time allowing relatively free wire translation. Preferably, the mass of 

c 

the rotating transducer parts is kept very small so that it only takes a small amount of 
friction to ensure coaxial rotation of wire 7 and disk 159 without slippage. Because die 
level of friction is so small, it does not substantially impede translational motion (Le., in- 
cut motion) of wire 7. Alternatively, wire 7 could include a lengthwise extending rib, slot 
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or flat, which is engaged with a corresponding key-way provided on disk 159. While more 
costly and difficult to manufacture, the use of a non-circular wire and correspo n ding key- 
way would ensure a slip-free connection between the wire and the disk, without introducing 

tran snatio nal friction* 

Hie principles of the present invention can also be applied to create an intravenous 
(IV) line insertion simulator. In the IV insertion simulator, a hardware device and software 
work together to reproduce the visual and physical experience of inserting an IV needle into 
an artery. The would work by having the user grasp a proxy needle 



to a 



ring device. Preferably, a custom IV hardware interface device would 
be designed for tins purpose. Alternatively, the Immersion PROBE could be used for 
performing a general simulation (without tactile fee dba ck). The insertion would be carried 
out by interacting with a simulation of die activity on a computer generated display. The 
display would show die needle as the user moves it along die part of the body into which 
the needle is to be inserted. As the needle is pushed into the skin, the simulation software 
would detect the contact and display an appropriate view of the process. For instance, if 
the needle were properly inserted into an artery, then a small amount of bleeding could be 
displayed along with a change in the shape of the skin where the needle has passed under. 
In addition, a farce mimicking that encountered by the user during an actual IV insertion 



could be generated by the IV interface device, in response to commands from the simi 
program, similar to the ar r angem ent cardiac catheterization simulator described in detail 
herein. 

The con c ep ts of die present invention are applicable to provide a realistic simulation 
of a wide range of other medical procedures, including particularly invasive surgical 
procedures wherein a device is inserted into an internal body passageway, or directly into 
body tissues. The procedures which can be simulated using die present inventive techniques 



1 to cardiology, invasive 







MM 


<- 1 



span every specialty of medicine, including but not lii 
radiology, neurosurgery, neurology, anesthesiology and, of course, general surgery. 
Procedures that can be simulated with die inventive system include, but are not limited to: 
• placement and/or use of therapeutic mechanical devices, e.g., 



stent 



I) r-.ifr - I I t - I 



i and coil placement for aneuri 



ii^niti^.Mii^i 



intravenous (TV) fine and central line placement, e.g., jugular, femoral. 
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• introduction into the body of materials or devices to enhance the ability to 
discern internal body parts, e.g., angiography, intra-coronary ultrasound 
devices, and fiber-optic scopes. 

• placement of devices to obtain tissue and fluid samples from the body, e.g., 
lumbar puncture, and liver and kidney biopsy. 

• needle placement, e.g., epidural, phlebotomy simulation, deployment of 
therapeutic agents at die cite of injury/disease (e.g., use of a catheter for 
precise placement of chemotherapeutic agents at die cite of an intravascular 
tumor). 

• placement and manipulation of devices to remove fluids or tissu e s, eg., 
urinary catheters, and pleural fluid removal. 

• devices for trauma management, e.g., chest tube placement 

• devices for airway management, e.g., fiber-optic laryngoscope, rigid 
laryngoscope, and intubation procedures. 

• devices for assessing disease and pathology, e.g., fiber-optic scope 
manipulation (e.g., cystoscqpes, bronchoscopy, flexible sigmoidoscopy, 
colonoscopy, and endoscopy). 

• devices for minimally invasive surgery, e.g., laparoscopic instrumentation. 
The present invention has been described in terms of preferred and exemplary 

embodiments thereof. Other embodiments, modifications and variations within the scope 
and spirit of the appended claims will, given the benefit of this disclosure, occur to persons 
of ordinary skill in the art. 
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1. 



A computer based catheterization simulation system, comprising: 
first memory means for storing image data representative of a virtual internal 
dy environment; 

second memory means for storing image data representing a virtual catheter 
ling within said internal body environment; 

display means for displaying images corresponding to said image data stored 
in said first and second memory means; 



i it ^ ii i* 



first input means for receiving an output signal from a first computer 
peripheral device that tracks die translational movement of an elongated relatively flexible 
r ep r e sen ting the virtual catheter, without a physical model corresponding to said 
internal body environment; 

first calculating means for calculating position data representing the position 
of the virtual Hflhf* 1 * 1 ' within said virtual internal body environment, based on die signal 
from the peripheral device; and 

first updating means for updating the image data representing the virtual 
rat ftrftr extending within said virtual internal body environment, based on said calculated 
position data. 

2, A catheterization simulation system according to claim 1, wherein: 
die end of said virtual catheter is angled; 

said peripheral device tracks the rotational movement of the elongated 
relatively flexible member and outputs to die first input means a signal based thereon; 

said first calculating means calculates orientation data representing an 



the first 



orientation of die angled end of die virtual catheter, based on die signal 
peripheral device; and said updating means updates the image data representing the virtual 
based on the calculated orientation data. 



w 




i 


hi 



g: 



l(<v(*4flllltHI 



hung means for determining the location of the virtual medical 



instrument within the virtual internal body environment; and 

output means for outputting a tactile feedback response signal to said first 
computer p j^ if h^rai device based on said location of die virtual catheter, said tactile 
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feed ba ck response signal serving to cause a mechanism of said first computer peripheral 
device to impart to said elongated relatively flexible member a tactile feedback force. 

4. A catheterization simulation system according to claim 3, wherein said tactile 
feedback force is a force resisting movement of said elongated relatively flexible member. 

5. A catheterization simulation system according to claim 1, further comprising: 
second calculating means for calculating the velocity of the virtual catheter 

within the virtual internal body environment, based upon the signal from the peripheral 
device; and 

second updating means for modifying the image data representative of the 
virtual internal body environment, based on the calculated velocity d 

6. A catheterization simulation system according to claim 5 , wherein the second 
updating means modifies the image data representative of the virtual internal body 
environment when the calculated velocity of the virtual catheter exceeds a preset velocity. 

7. A surgical catheterization simulation system according to claim 6, wherein 
the second updating means modifies the image data representative of the virtual internal 
body environment to simulate a puncture of an internal body passageway, and associated 
bl eed ing 

8. A catheterization simulation system according to claim 1, further comprising 
third memory means for storing image data representing a simulated real-time wave form 
of a signal representing a physiological parameter, and wherein said display means displays 
said simulated real-time wave form together with said images corresponding to said image 
data stored in said first and second memory means. 

9. A catheterization simulation system according to claim 8, wherein said 
simulated real-time waveform is an electrocardiogram (EKG) waveform. 

10. A catheterization simulation system according to claim 9, further comprising 
second updating means for modifying the image data representative of the virtual internal 
body environment, based on the image data representing the EKG waveform. 

11. A catheterization simulation system according to claim 10, wherein the 
virtual internal body environment comprises a vascular passageway, and the image data 
modification based on the simulate real-time waveform simulates a pulsation of the 
vascular passageway in synchronism with the EKG waveform. 

12. A catheterization simulation system acconiing to daim 1, further comprising: 
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a second input 



s for receiving a signal from a second 



uter 



1 device; 




device* 



13. A catheterization simulation system according to claim 12, wherein the 
virtual internal body environment comprises a vascular passageway, and said image data 
m odificatio n based on die signal from the second computer peripheral device simulates the 
release and flow of contrasting dye through the vascular passageway* 



said second computer peripheral device. 

15. A catheterization simulation system according to claim 14, wherein die 
virtual internal body environment comprises a vascular passageway having an occlusion 
therein, and said image data modification based on the signal from the second computer 
peripheral device simulates the inflation of a balloon catheter within said occlusion. 

16. A catheterization simulation system according to claim IS, wherein an 
inflation amount of the simulated balloon catheter increases with time while a switch of die 
second computer peripheral device is held in an ON position. 

17. A Catheterization omiilfltinn system aranrriing tn claim 1 6, farther comprising 

third updating means for modifying the image data representative of the virtual vascular 
passageway, based on the inflation amount of the simulated balloon catheter. 

18. A catheterization simulation system according to claim 17, wherein die image 
data modification based on die inflation amount of the simulated balloon catheter simulates 
die dearing of die occlusion. 

19. A catheterization simulation system according to claim 16, wherein die image 
data mnrfifirfltinn based »» the signal from die second c ompute r peripheral device simulat e s 

die rupture of die simulated balloon catheter in the event die switch is held in the ON 
position for a time period exceeding a preset time limit. 




peripheral device; and 



catheter 
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20. A catheterization simulation system according to claim 1, wherein said image 
data representative of a virtual internal body environment, and said image data representing 
a virtual catheter, comprise 3-D image data. 

21. A catheterization simulation system according to claim 11, wherein said 
vascular passageway comprises an arterial tree. 

22. A catheterization simulation system according to claim 13, wherein said 
vascular passageway comprises an arterial tree. 

23. A catheterization simulation system according to claim 15, wherein said 
vascular passageway co mpri ses an arterial tree. 

24. A computer based medical procedure simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 
body environment; 

second memory means for storing image data representing a virtual medical 
instrument extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 

in said first and second memory means; 

first input means for receiving an output signal from a first computer 
peripheral device that tracks the movement of a physical member representing the virtual 

medical instrument; 

first calculating means for calculating position data representing the position 
of die virtual medical instrument within said virtual internal body environment, based on 
the signal from the peripheral device; 

first updating means for updating the image data representing the virtual 

medical instrument extending within said virtual internal body environment, based cm said 

calculated position data; 

second calculating means for calculating the velocity of die virtual medical 

instrument within the virtual internal body environment, based upon die signal from the 

peripheral device; and 

second updating means for modifying the image data representative of the 
virtual internal body environment, based cm the calculated velocity data. 

25. A medical procedure simulation system according to claim 24, wherein the 
second updating means modifies the image data representative of die virtual internal body 
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environment when the calculated velocity of the virtual medical instrument exceeds a preset 
velocity. 

26. A medical procedure simulation system according to claim 25, wherein the 
second updating means modifies the image data representative of the virtual internal body 
environment to simulate a puncture of an internal body passageway, and associa te d 
bleeding. 

27. A computer based medical procedure simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 
body environment; 

second memory means for staring image data representing a virtual medical 
instrument extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 

in said first and second memory means; 

first input means for receiving an output signal from a first computer 
peripheral device that tracks the movement of a physical member representing die virtual 

medical instrument; 

first calculating means for calculating position data representing the position 
of the virtual medical instrument within said virtual internal body environment, based on 
the signal from the peripheral device; 

first updating means for updating the image data representing the virtual 
medical instrument extending within said virtual internal body environment, based on said 

calculated position data; and 

second updating means for modifying die image data representative of the 
virtual internal body environment, based on die signal from said peripheral device, to 
simulate a puncture of an internal body passageway, and asso ci ated bleeding. 

28. A computer based medical procedure simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 

body environment; 

s eco nd memory means for storing image data representing a virtual medical 
instrument extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 

in said first and second memory means; 
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first input means for receiving an output signal from a first computer 
peripheral device that tracks the movement of a physical member representing die virtual 

medical instrument; 

first calculating means for calculating position data representing the position 
of the virtual medical instrument within said virtual internal body environment, based cm 
the signal from the peripheral device; 

first updating means for updating die image data representing the virtual 

medical instrument extending within said virtual internal body environment, based on said 

calculated position data; and 

third memory means for storing image data representing a simulated real-time 

wave form of a signal representing a physiological parameter; 

wherein said display means displays said simulated real-time wave form 

together with said images corresponding to said image data stored in said first and second 

memory means. 

29. A medical procedure simulation system according to claim 28, wherein said 
simulated real-time waveform is an electrocardiogram (EKG) waveform. 

30. A medical procedure simulation system according to claim 28, further 
comprising second updating means for modifying the image data representative of the 
virtual internal body environment, based on the image data representing the simulated real- 
time waveform. 

31. A medical procedure simulation system according to claim 29, further 
comprising second updating means for modifying the image data representative of die 
virtual internal body environment, based on the image data representing the simulated EKG 
waveform. 

32. A medical procedure simulation system according to claim 31, wherein die 
virtual internal body environment comprises a vascular passageway, and the image data 
modification based on the simulated real-time waveform simulates a pulsation of the 
vascular passageway in synchronism with the EKG waveform. 

33. A medical procedure simulation system according to claim 32, wherein said 
vascular passageway comprises an arterial tree. 
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34. A medical procedure simulation system according to claim 29, wherein the 
image data representing the EKG wave form includes EKG irregularity data for generating 

periods of EKG irregularity. 

35. A medical procedure simulation system according to claim 34, wherein said 
EKG irregularity is randomly generated. 

36. A medical procedure simulation system according to claim 28, further 
comprising second updating means for modifying the image data representing the simulated 
real-time waveform based upon a change in the image data representative of die virtual 
internal body environment. 

37. A medical procedure simulation system according to claim 29, further 
comprising second updating means for modifying the image data representing die EKG 
waveform based upon the existence or non-existence of data representing a simulated 
vascular abnormality in die image data representative of die virtual internal body 
environment. 

38. A medical procedure simulation system according to claim 37, wherein the 
simulated vascular abnormality comprises a vascular occlusion. 

39. A computer based medical procedure simulation system, comprising: 
first memory means for storing image data representative of a virtual internal 

body environment; 

second memory means for storing image data representing a virtual medical 

instrument extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 

in said first and second memory means; 

first input means for receiving an output signal from a first computer 
peripheral device that tracks die movement of a physical member representing the virtual 

medical instrument; 

first calculating means for calculating position data representing the position 
of die virtual medical instrument within said virtual internal body environment, based on 
the signal from die peripheral device; 

first updating means for updating the image data r epresenting die virtual 
medical instrument extending within said virtual internal body environment, based on said 
calculated position data; and 
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second updating means for modifying die image data representative of the 
virtu al internal body environment so as to simulate the infusion of one fluid into another. 

40. A medical procedure simulation system according to claim 39, wherein the 
simulated infusion of one fluid into another comprises simulated bleeding based on a 
calculated velocity of die virtual medical instrument 

41. A medical procedure simulation system according to claim 40, wherein die 
velocity of die virtual medical instrument is calculated from said calculated position data. 

42. A computer based medical procedure simulation system, comprising: 
first memory means for storing image data representative of a virtual internal 

body environment; 

display means for displaying images corresponding to said image data stored 

in said first memory means; and 

updating means for modifying the image data representative of the virtual 
internal body environment so as to simulate the infusion of one fluid into another. 

43. A medical procedure simulation Systran according to claim 42, wherein die 
simulated infusion of one fluid into another comprises a simulated infusion of contrasting 

dye into a bloodstream. 

44. A medical procedure simulation system according to claim 42, wherein the 
simulated infusion of one fluid into another is accomplished by streaming into a stored 
infusion value definition a ramp constrained random flow of values, to thereby vary the 
concentration of said one fluid at various points. 

45. A medical procedure simulation system according to claim 43, wherein the 
simulated infusion of contrasting dye into a bloodstream is accomplished by streaming into 
a stored infusion value definition a ramp constrained random flow of values, to thereby 
vary the concentration of die dye at various points. 

46. A co m p u t e r based catheterization simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 

body environment; 

second memory means for storing image data representing a virtual catheter 

extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 
in said first and second memory means; 
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first input means for receiving an output signal from a first computer 
peripheral device thai tracks the translaiional movement of an elongated relatively flexible 
member representing the virtual catheter; 

first calculating means for calculating position data representing the position 
of die virtual catheter within said virtual internal body environment, based on die signal 
from die peripheral device; 

first updating means for updating the image data representing the virtual 
ca fhftw extending within said virtual internal body environment, based on said calculated 
nosition Aita* 

a second input means for receiving a signal from a second computer 

peripheral device; and 

second updating means for modifying the image data representing the virtual 
catheter extending within said virtual internal body environment based on the signal from 
said second comp u ter peripheral device; 

wherein die virtual internal body environment comprises a vascular passageway 
having an occlusion therein, and said image data modification based on the signal from die 
second computer peripheral device simulates the inflation of a balloon catheter within said 
occlusion. 

47. A catheterization simulation system according to claim 46, wherein an 
inflation amount of the irirmilafrrf balloon catheter increases with time while a switch of die 
second computer peripheral device is held in an ON position. 

48. A rathrtfr riratiri n simulation system according to claim 47 r further comprising 

third updating means for modifying die image data representative of the virtual vascular 
passageway, based cm the inflation amount of die simulated balloon catheter. 

49. A catheterization wm iilatitt n system according to claim 48, wherein the image 

Aita modification based on the inflation amount of the simulated balloon catheter simulates 
die clearing of die occl u s ion . 

50. A catheterization simulation system according to claim 47, wherein the image 
data modification based on the signal from the second computer peripheral device simulates 
the rupture of the siTm i1afffH balloon catheter in the event the switch is held in the ON 
position for a time period exceeding a preset time limit. 
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